1. Field of the Invention
The present invention relates to an obstacle detection device for detecting any obstacles on a road without confusing those with textures using a plurality of cameras mounted in a vehicle or provided above the road, and a method applied to the device. Here, the obstacles are those located on the road such as other vehicles ahead, parked vehicles, and pedestrians. The textures include white lines, road signs, paint, road stains, and shadows of roadside objects, all of which do not disturb vehicle driving. The cameras are provided mainly for helping safe driving and realizing automatic driving of the vehicle, or for counting the number of passing vehicles on the road or monitoring those passing vehicles for their driving. Obstacle detection is possible even if the relationship between the road plane and the respective cameras constantly changes in relative position or posture due to camera vibration or a change in road tilt, for example.
2. Description of the Background Art
A method for detecting any obstacles on roads is classified into two types: one is a type using active sensors typified by sonars, radars, and range finders; and the other is a type using passive sensors typified by visible-light CCD cameras and infrared cameras.
The active sensors are popular for measuring object positions in various applications, and well known for their usability. The active sensors have, however, problems for an application of detecting any obstacles lying in the vehicles' way on the roads such as other vehicles. Specifically, the problems are associated with low detection resolution, not enough measurement range, erroneous detection of non-obstacles on the roads, and erroneous detection of objects lying on non-disturbing road side due to no driving lane detection capability. Thus, there has been a demand for an advanced obstacle detection technology by image analysis using the passive sensors exemplified by CCD cameras.
To detect obstacles lying on road surfaces through analysis of images provided exemplarily by CCD cameras mounted in vehicles, generally utilized is information about image brightness intensity pattern or driving lanes recognized for the purpose. As for detecting the driving lanes, cutting out parts in shades of gray with less texture will do from images picked up by a camera.
The issue here is that, many obstacles are actually similar to roads in brightness intensity or pattern, resulting in difficulty achieving the higher usability with the less erroneous detection.
There is another type of method using a plurality of cameras for detecting obstacles and driving lanes. Such a method is generally called a stereoscopic method.
With stereoscopic views, three-dimensional (3D) information about a target detection region can be derived on the triangulation principle. Thus, stereoscopic views seem to be a solution for obstacle and lane detection with higher accuracy, but still bear problems. For example, a corresponding point search cannot be uniquely solved, and the calculation cost is quite expensive. This corresponding point search is done to find any specific point(s) in real world shared by a plurality of camera images.
In this respect, methods disclosed in Patent Literature 1 (JP-A-2000-293693) and Patent Literature 2 (JP-A-2001-76128) do not require such a corresponding point search, and are considered useful for obstacle detection. These methods are described in the below.
Assuming now that two cameras, right and left, are provided to pick up images of a road. Project points as a result of projecting points on the road plane onto images picked up by the right and left cameras are presumably (u, v) and (u′, v′), and a relational expression 1 is established as follows:
                                          u            ′                    =                                                                      h                  11                                ⁢                u                            +                                                h                  12                                ⁢                v                            +                              h                13                                                                                      h                  31                                ⁢                u                            +                                                h                  32                                ⁢                v                            +                              h                33                                                    ,                                  ⁢                              v            ′                    =                                                                      h                  21                                ⁢                u                            +                                                h                  22                                ⁢                v                            +                              h                23                                                                                      h                  31                                ⁢                u                            +                                                h                  32                                ⁢                v                            +                              h                33                                                                        (        1        )                                h        =                  (                                    h              11                        ,                          h              12                        ,                          h              13                        ,                          h              21                        ,                          h              22                        ,                          h              23                        ,                          h              31                        ,                          h              32                        ,                          h              33                                )                                    (        2        )            
The equation 2 shows a parameter dependent on the positions and postures of the right and left cameras with respect to the road plane, lens focal distances of the cameras, points of origin of the images, and the like. The parameter h can be derived in advance only by project points (ui, vi) and (ui′, vi′) (i=1, 2, . . . , N), which are those derived by projecting four or more points on the road plane onto the right and left images. Using such a relational expression, a corresponding point P′(u′, v′) on the left image is derived based on the assumption that an arbitrary point P(u, v) on the right image is located on the road plane. If the point P is truly located on the road plane, the points P and P′ are paired as the correct corresponding points, leading to a good match between two pixels or neighboring regions in terms of brightness intensity or feature. On the other hand, if the points P and P′ differ in brightness intensity, the point P is determined as belonging to an obstacle region. This method allows for determining whether an arbitrary point in the image has a height from the road plane directly only from the relational expression 1. There is thus no need for the corresponding point search between the right and left images.
To apply such a scheme for obstacle detection in front of the vehicle, the parameter h is presumed as roughly constant when the vehicle is driving on rather flat road at low speed. Thus, there is no need to calculate the parameter h twice for correct obstacle detection.
Here, as to correspondence detection, the operation of a section provided therefor is described by referring to FIGS. 3 to 5.
The correspondence detection section operates to convert a first image picked up by a first image pick-up device into an image viewed from a viewpoint of a second image pick-up device. A parameter used for this conversion is so calculated as to keep a typical geometric relationship between a plurality of image pick-up devices and the road plane, with a presumption that the vehicle is standing still on the no-tilting road plane. The parameter is not calculated twice, and not changed during obstacle detection, e.g., when the vehicle is moving.
The parameter is calculated in a manner based on the Patent Literature 1, and described in the below.
Referring to FIG. 3, two cameras a and b are set up. The road surface has two parallel white lines l and l′ extending roughly along the optical axes of the cameras. The obstacle detection device is not notified of the relationship between the two cameras a and b in position and posture, but only of epipolar constraint. During when the obstacle detection device is in operation, no change occurs, presumably, to the relative positions and postures of the cameras a and b, and epipolar constraint. Here, the epipolar constraint means a constraint condition for stereoscopic images of a general type. Under this condition, as shown in FIG. 4, the arbitrary point P on the image (right image) picked up by the camera a is so constrained as to be on a predetermined linear line including the corresponding point P′ on the image (left image) picked up by the camera b. This linear line is referred to as an epipolar line. As an example, when the optical axes of the cameras are so placed as to be parallel to each other, the corresponding point of the arbitrary point P in the right image is found on the same scanning line on the right image. Accordingly, the epipolar line agrees with the scanning line. The epipolar constraint is dependent on the relationship between the stereoscopic cameras in relative position and posture, and internal parameters of the cameras, e.g., lens focal distance, origin point of images. Thus, the epipolar constraint being invariant means the relative positional relationship between the stereoscopic cameras and their internal parameters showing no change (during when the obstacle detection device is in operation or the vehicle having the device mounted therein is moving). This epipolar constraint is formulated as the following equation 3.(u,v,1)F(u′,v′,1)T=0  (3)
Herein, (u, v) is the arbitrary point P on the right image, and (u′, v′) is the corresponding point of the point P on the left image. F denotes a 3×3 matrix, and referred to as Fundamental matrix. Expanding the equation 3 will lead to the following equation 4.(F11u+F12v+F13)u+(F21u+F22v+F23)v′+(F31u+F32v+F33)=0  (4)
Herein, Fji (i, j=1, 2, 3) denotes an element of j row(s) and i column(s) of the matrix F, and can be derived from a plurality of corresponding points. Further, the equation 4 denotes an epipolar line corresponding to the point P(u, v) on the right image. Nine elements of the matrix F are not all independent, and theoretically, are derivable from seven corresponding points. Because 3D position is not required for each pair of the corresponding points, calculating the matrix F, i.e., the epipolar constraint, is rather easy. The lines l and l′ in each image are parallel three-dimensionally but not on the images picked up by the right and left cameras. As shown in FIG. 5, the lines l and l′ in each image cross each other at a point at infinity, which is called a vanishing point. Next, derived is a relationship established between the corresponding points on the road plane. As shown in the right image of FIG. 5, arbitrary points on the linear line l are P1 and P3, and arbitrary points on the linear line l′ are P2 and P4. For these four points, corresponding points P1′, P2′, P3′, and P4′ in the left image can be calculated using the epipolar constraint previously derived. That is, the point P1′ correspond to the point P1 agrees with an intersection point of the linear line l and the epipolar line L1 of the point P1 on the left image. Similarly, the points P2′, P3′, and P4′ can be derived as intersections, respectively, of the epipolar lines L2, L3, and L4 of the points P2, P3, and P4, and the linear line l or l′. Assuming that coordinates of the point Pi (i=1, 2, 3, 4) are (ui, vi), and coordinates of the point Pi′ (i=1, 2, 3, 4) are (ui′, vi′). The relation between the coordinates (ui, vi) and (ui′, vi′) can be expressed by a relational expression 5.
                                                        u              i              ′                        =                                                                                h                    11                                    ⁢                                      u                    i                                                  +                                                      h                    12                                    ⁢                                      v                    i                                                  +                                  h                  13                                                                                                  h                    31                                    ⁢                                      u                    i                                                  +                                                      h                    32                                    ⁢                                      v                    i                                                  +                                  h                  33                                                              ,                                          ⁢                                    v              i              ′                        =                                                                                h                    21                                    ⁢                                      u                    i                                                  +                                                      h                    22                                    ⁢                                      v                    i                                                  +                                  h                  23                                                                                                  h                    31                                    ⁢                                      u                    i                                                  +                                                      h                    32                                    ⁢                                      v                    i                                                  +                                  h                  33                                                                    ⁢                                  ⁢                  (                                    i              =              1                        ,            2            ,            3            ,            4                    )                                    (        5        )            
These eight equations are solved using the following equation 6.h=(h11,h12,h13,h21,h22,h23,h31,h32,h33)  (6)
If an arbitrary solution h satisfies the equation 5, a constant multiple kh of h (k is constant) also satisfies the equation 5. No generality is thus lost with h33=1, and eight equations will lead to h composed of nine elements. By using such derived h, the corresponding point P′(u′, v′) on the right image can be calculated as the following equation 7 with an assumption that the arbitrary point P(u, v) on the left image is located on the road plane.
                                          u            ′                    =                                                                      h                  11                                ⁢                u                            +                                                h                  12                                ⁢                v                            +                              h                13                                                                                      h                  13                                ⁢                u                            +                                                h                  32                                ⁢                v                            +                              h                33                                                    ,                                  ⁢                              v            ′                    =                                                                      h                  21                                ⁢                u                            +                                                h                  22                                ⁢                v                            +                              h                23                                                                                      h                  31                                ⁢                u                            +                                                h                  32                                ⁢                v                            +                              h                33                                                                        (        7        )            
With the methods in Patent Literatures 1 and 2, when the vehicle drives on typical outside roads, the relationship between the road plane and the respective cameras continuously changes in relative position and posture due to vibrations occurring to the obstacle detection device, or a change in road tilt. Consequently, these methods bear such problems, due to vehicle vibration, as frequent erroneous detection especially around the texture on the road plane such as white lines, road signs, paint, road stains, shadows of roadside objects and vehicles, and the like.
As described above, with an obstacle detection device using CCD cameras of a conventional type, usage environment is limited, or the relationship between the road plane and the respective cameras continuously changes in relative position and posture due to vibrations during the device operation or driving vehicle. As a result, frequent erroneous detection occurs especially around the texture on the road plane such as white lines, road signs, paint, road stains, shadows, and the like, considerably lowering the true detection accuracy of obstacle detection.
The present invention is proposed in consideration of the above conventional problems, and an object thereof is to provide an obstacle detection device capable of correctly detecting only true obstacles no matter what road the device is set up, or no matter what road a vehicle having the device mounted therein is driving.